Monitoring device, suspicious object detecting method, and recording medium

ABSTRACT

A monitoring device and the like are provided which are capable of detecting an attribute change in a suspicious object that cannot be determined from the behavior of the object. An associating unit associates, among a plurality of objects detected from time-series image data, identical objects with one another. An attribute change detecting unit detects from the time-series image data a change in an attribute of at least one of the identical objects and an attendant item. A suspicious object detecting unit detects a suspicious object on the basis of the change in attribute.

TECHNICAL FIELD

The present invention relates to a monitoring device, a suspiciousobject detecting method, and a recording medium, and particularly to amonitoring device that detects a suspicious object.

BACKGROUND ART

A related technique, for detecting a suspicious object based on thebehavior of an object detected from time-series images, exists. PTL 1describes that the behavior of a person appearing in a monitoring videois detected, and the detected behavior of the person is matched with apattern of suspicious behavior. A “degree of suspicion” (level ofsuspicion) is associated with each pattern of suspicious behavior. Thetechnique described in PTL 1 outputs an alarm when the “degree ofsuspicion” of a person exceeds a threshold.

PTLs 2, 3, and 4 describe a technique for tracking a person by matchinga feature of a specified person with the feature of a person detectedfrom time-series images using at least one of a feature of the person'sface and a feature of the person's appearance, and using the matchingresult.

Specifically, PTLs 2 and 3 describe a technique for tracking a person bynewly adding a feature of clothing when the clothing is changed whiletracking the person. Meanwhile, PTL 4 describes a technique for trackinga person to detect a shoplifting action. Here, it is described that achange such as bulging of a pocket of the clothing or fast walking aftershoplifting is detected to determine whether the person to be tracked issuspicious.

CITATION LIST Patent Literature

[PTL 1] JP 2011-107765 A

[PTL 2] JP 2014-16968 A

[PTL 3] JP 2016-127563 A

[PTL 4] JP 2016-057908 A

SUMMARY OF INVENTION Technical Problem

In the related technique described in PTL 1, for example, it is notpossible to detect that a human (object) has disguised himself/herselfor has left a carried object in a place not shown in a monitoring video.

In the techniques described in PTLs 2 and 3, since the clothing changeitself is not determined to be suspicious, it is not possible to extracta person who has changed his/her clothing as a suspicious person andoutput an alert. The technique described in PTL 4 detects only thechange of clothing or gait of a person within the range of samenessassumed for a specific action such as shoplifting. For this reason, itis not possible to detect the switching of clothing or gait of theperson itself and output an alert for a suspicious person.

The present invention has been made in view of the above problems, and amain object thereof is to provide a monitoring device and the likecapable of detecting a suspicious object that cannot be determined fromthe behavior of the object.

Solution to Problem

A monitoring device according to an aspect of the present inventionincludes: association means for associating the same object amongmultiple objects detected from time-series image data; attribute changedetection means for detecting an attribute change of at least one of thesame object and an accessory of the object from the time-series imagedata; and a suspicious object detection means for detecting a suspiciousobject based on the attribute change.

A suspicious object detecting method according to an aspect of thepresent invention includes: associating the same object among multipleobjects detected from time-series image data; detecting an attributechange of at least one of the same object and an accessory of the objectfrom the time-series image data; and detecting a suspicious object basedon the attribute change.

A recording medium according to an aspect of the present inventionrecords a program for causing a computer to execute processing of:associating the same object among multiple objects detected fromtime-series image data; detecting an attribute change of at least one ofthe same object and an accessory of the object from the time-seriesimage data; and detecting a suspicious object based on the attributechange.

Advantageous Effect of Invention

According to one aspect of the present invention, it is possible todetect a suspicious object that cannot be determined from the behaviorof the object.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a configuration of a monitoringdevice according to Example embodiment 1.

FIG. 2 is a flowchart illustrating an operation flow of the monitoringdevice according to Example embodiment 1.

FIG. 3 is a block diagram illustrating a configuration of a monitoringdevice according to Example embodiment 2.

FIG. 4 is a flowchart illustrating an operation flow of the monitoringdevice according to Example embodiment 2.

FIG. 5 is a block diagram illustrating a configuration of a monitoringdevice according to Example embodiment 3.

FIG. 6 is a flowchart illustrating an operation flow of the monitoringdevice according to Example embodiment 3.

FIG. 7 is a block diagram illustrating a configuration of a monitoringdevice according to Example embodiment 4.

FIG. 8 is a table illustrating an example of exception information.

FIG. 9 is a flowchart illustrating an operation flow of the monitoringdevice according to Example embodiment 4.

FIG. 10 is a block diagram illustrating a configuration of a monitoringdevice according to Example embodiment 5.

FIG. 11 is a diagram illustrating a hardware configuration according toExample embodiment 6.

EXAMPLE EMBODIMENT Example Embodiment 1

Example embodiment 1 will be described with reference to FIGS. 1 and 2.

(Monitoring Device 100)

FIG. 1 is a block diagram illustrating a configuration of a monitoringdevice 100 according to Example embodiment 1. As illustrated in FIG. 1,the monitoring device 100 includes an association unit 110, an attributechange detection unit 120, and a suspicious object detection unit 130.

(Association Unit 110)

The association unit 110 associates the same objects with each otheramong multiple objects detected from time-series image data. Theassociation unit 110 is an example of association means. For example,the association unit 110 associates identification information (e.g.,label added to each object) of the same objects and stores them in amemory (not illustrated).

An object is a body to be monitored in image data. For example, anobject is a moving object (person, vehicle, or the like) or an installedobject. Attribute information of an object is information regarding theattribute of the object. The attribute of an object is a feature uniqueto the object and forms the modality of the object. The modality refersto an aspect, an atmosphere, or a way sensed by the five human senses. Aunique feature referred to herein only needs to be unique within theobservation target period, and does not need to be semi-permanentlyunique.

In particular, when the object is a person, the attribute of the objectis one or more features that form the apparent personality of theperson, and specifically includes at least one of the face, iris, gait,body shape, and hairstyle of the person.

Features of accessories of an object are also included in the attributeof the object. An accessory is another object attached to a certainobject. For example, when the object is a person, the accessory is abag, a suitcase, baggage, clothing, footwear, an accessory (e.g., mask,eyeglass, hat, or the like), or other belongings. Further, an accessoryalso includes a portion of the person integrated with the accessory,such as the face or hairstyle of the person. Further, the attribute ofthe accessory itself may also be a part of the attribute of the object.The attribute of the accessory includes at least one of color, size,shape, pattern, number, position, and texture of the accessory.

Some accessories, such as clothing, are detected without beingdistinguished from the object, and other accessories, such asbelongings, are detected while being distinguished from the object.Alternatively, an accessory closely attached to the object, such as abackpack carried on the back by a person (example of object), may bedetected together with the object (i.e., integrally). Here, regardlessof whether the object and the accessory are inseparably detected, thefeature of the accessory is treated as a part of the attribute of theobject.

In the case of an accessory detected separately from the object, a bodythat is in the vicinity of the object (e.g., within predetermined rangefrom position of object) and moves along with the object is detected asan accessory of the object.

The association unit 110 associates the same objects with each otheramong the multiple objects detected from the time-series image data, onthe basis of at least one of attribute information and positioninformation of the objects. Hereinafter, first to third examples ofassociation of the same object by the association unit 110 will bedescribed.

In a first example, the association unit 110 associates the same objectswith each other among the multiple objects, on the basis of positioninformation of the objects. Position information of an object includesposition coordinates of a region of the object in each piece of imagedata. For example, an object can be tracked by associating the sameobject using position information of the object detected fromtime-series image data. A method of tracking an object is described inPTLs 2 to 4, for example.

The position coordinates may be position coordinates on an image, or maybe a real-world coordinate system obtained by converting the positioncoordinates on an image by a camera parameter representing the positionand pose of a camera. The camera parameter can be calculated byperforming calibration in advance. When tracking an object, theassociation unit 110 may use information regarding the similarity in theappearance of the object in addition to the attribute of the object. Forexample, the association unit 110 may also use the similarity in thevisual features (color, pattern, shape, and the like) of the image inthe region of the object, the similarity in the motion, and the like fortracking the object.

In a second example, the association unit 110 associates the sameobjects with each other among the multiple objects, on the basis of theattribute information of the objects. In this case, the association unit110 determines the sameness of the object by using the attributeinformation of the object, and further outputs the samenessdetermination result of the object. For example, the association unit110 calculates a score based on the similarity or distance betweenpieces of attribute information of persons as a scale for determiningthe sameness of an object.

In particular, when the object is a person, the association unit 110 mayassociate the same person by identifying the person by face matching,iris matching, or matching using other biometric information amongmultiple persons detected in the time-series image data. The samenessdetermination of the person may be performed by identification means(not illustrated). In this case, the association unit 110 associates thesame person on the basis of the result of the sameness determination ofthe person by the identification means.

When the attribute information includes multiple elements, theassociation unit 110 may associate the same object by using multiplepieces of attribute information in combination. In this case, theassociation unit 110 weights the object sameness determination resultsthat are based on the multiple pieces of attribute information,according to the accuracy of the object sameness determination madebased on each attribute information. For example, the association unit110 adds a coefficient between 0 and 1 for a likelihood (e.g., 30%chance of being object A and 70% chance of being object B) indicatingthe sameness determination result of the object.

The association unit 110 integrates the weighted sameness determinationresults. One method of weighting is to determine the weight on the basisof the accuracy of the object sameness determination made based on eachattribute information. For example, the object sameness determinationtest made based on each attribute information is repeated, and theaccuracy of the object sameness determination made based on theattribute information is determined according to the number or the ratioof correctly determined times. The association unit 110 weights theobject sameness determination results that are based on the individualattribute information, and then adds the results. Note, however, thatthe integration method is not limited to this. Alternatively, theassociation unit 110 can use any method in order to integrate thesameness determination results for the multiple attributes. Theassociation unit 110 associates the same object on the basis of theintegrated sameness determination result.

In a third example, the association unit 110 determines the sameness ofan object and associates the same object by using both attributeinformation and position information of the object. In this case, theassociation unit 110 weights each of the object sameness determinationresult of that is based on the attribute information and the objectsameness determination result that is based on the position information.Then, the association unit 110 associates the same object on the basisof a result of integrating the two weighted sameness determinationresults. Alternatively, as described above, the association unit 110 mayintegrate the two sameness determination results by another method.

The attribute information is not necessarily extracted from every pieceof image data. When attribute information cannot be extracted from imagedata, as described in the first example, the association unit 110determines the sameness of the object by tracking based on the positioninformation. In this manner, the criterion of the sameness determinationmay vary among time-series images. As a result, the reliability of thesameness determination also varies. Hence, the association unit 110 mayinclude, in the association information, information indicating thereliability of the sameness determination for each image.

The reliability of the association also depends on whether other objectsexist around the object and whether attributes of the objects aresimilar to each other. Hence, the association unit 110 may obtain thereliability in consideration of the distribution status of othersurrounding objects.

The association unit 110 associates the same object among the objectsdetected from the time-series image data by any of the methods describedabove. That is, when the score indicating the sameness between objectsis equal to or more than a certain value, the association unit 110determines that the objects are the same and associates the objects witheach other. Then, the association unit 110 groups the associated sameobjects. Attribute information of the associated same objects is alsogrouped.

As described above, since the attribute information cannot always beextracted, the attribute information is sometimes extractedintermittently in time series. That is, a situation occurs in which onlya part of the attribute information is extracted at a certain time, andthe rest of the attribute information is also extracted at another time.The association unit 110 associates the attribute information extractedin this way, too, with the group determined to be the same object.

The association unit 110 transmits sameness determination information ofthe associated objects to the attribute change detection unit 120. Theobject sameness determination information includes information fordistinguishing a group of objects determined to be the same by theassociation unit 110 and attribute information of the objects associatedwith the group. The information for distinguishing a group of objectsdetermined to be the same by the association unit 110 is also referredto as a group ID (identification). Further, the object samenessdetermination information may also include position information of theobjects.

As illustrated in the following cases 1, 2, and 3, the association unit110 may indirectly associate the same person (example of object) amongobjects detected from time-series image data, by also using informationsuch as a distribution status of other surrounding objects andconstraints in the environment. In these cases, the association unit 110associates the same person using at least one of the attributeinformation of the object and the position information of the object andalso using information regarding the environment or the like.

Case 1: No other person exists around the detected person.

Case 2: All other persons are identified, and only one unidentifiedperson exists.

Case 3: In a situation where the room has one doorway and there was noother person in the room, one person enters and then leaves the room.

(Attribute Change Detection Unit 120)

The attribute change detection unit 120 detects a change in theattribute of at least one of the same object and accessory from thetime-series image data. That is, the attribute change detection unit 120detects, among the objects determined to be the same, an object in whichsome attributes (including attribute of accessory) do not indicatesameness. The attribute change detection unit 120 is an example ofattribute change detection means.

Specifically, the attribute change detection unit 120 receives theobject sameness determination information from the association unit 110.The attribute change detection unit 120 groups the same objects in thetime-series image data by using the object sameness determinationinformation and further checks consistency of attributes of the sameobjects.

More specifically, the attribute change detection unit 120 detects achange in the attribute by calculating the similarity between attributesof the same objects specified in the time-series image data.Hereinafter, a change in the attribute of an object or an accessory maybe simply referred to as an attribute change.

For example, the attribute change detection unit 120 detects, as anattribute change, a change in at least one of the color, size, shape,pattern, number, position, and texture of the accessory of the sameobject between pieces of time-series image data. Note, however, that theattribute change described here is an example. The attribute changedetection unit 120 may detect any attribute change of an object or anaccessory.

For example, when the object is a person and a person determined to bethe same from the gait information and position information puts on a 3Dmask at some point so that the person's face changes, this may also bedetected as an attribute change by the attribute change detection unit120. Similarly, when a person changes his/her clothing so that theattribute of the clothing changes to the sameness state, this may alsobe detected as an attribute change by the attribute change detectionunit 120.

Specifically, the attribute change detection unit 120 detects, as anattribute change, the fact that the similarity between attributes ofobjects determined to be the same associated in time series falls belowa certain threshold (value lower than threshold used in samenessdetermination). Note, however, that the similarity between theattributes may temporarily decrease due to factors such as anenvironmental change. For this reason, after the similarity between theattributes falls below a certain threshold, the attribute changedetection unit 120 may detect the attribute change after confirming thatthe attribute after the change of the object is stable. That is, theattribute change detection unit 120 may detect the attribute change whena state in which the similarity of the changed attribute is equal to ormore than a certain threshold continues for a certain period or longer.

When an attribute change is detected, the attribute change detectionunit 120 transmits an attribute change detection result to thesuspicious object detection unit 130. An attribute change detectionresult includes a group ID of a group that identifies the object, thetype of the changed attribute, and information indicating the content ofthe attribute change.

(Suspicious Object Detection Unit 130)

The suspicious object detection unit 130 detects a suspicious object onthe basis of a change in attribute. The suspicious object detection unit130 is an example of suspicious object detection means.

Specifically, the suspicious object detection unit 130 receives anattribute change detection result of the same object from the attributechange detection unit 120.

The suspicious object detection unit 130 detects the object in which theattribute change is detected as a suspicious object.

The suspicious object detection unit 130 may give notice that asuspicious object has been detected. For example, the suspicious objectdetection unit 130 may output an alert or may notify a terminalpossessed by a security guard.

In one modification, the suspicious object detection unit 130 maycalculate the “degree of suspicion” that is a numerical valuerepresenting the degree (level) of suspicion of the object by matchingthe detected attribute change of the object with a predeterminedattribute change pattern. For example, when the attribute of only theface or clothing is changed, the suspicious object detection unit 130may increase the value of the degree of suspicion. Then, the “degree ofsuspicion” may be accumulated in time series, and when the accumulatedvalue exceeds a threshold, the suspicious object detection unit 130 mayoutput an alert.

(Operation Flow of Monitoring Device 100)

An operation flow of the monitoring device 100 according to Exampleembodiment 1 will be described with reference to FIG. 2. FIG. 2 is aflowchart illustrating an operation flow of the monitoring device 100.

As illustrated in FIG. 2, the association unit 110 associates the sameobjects with each other among multiple objects detected from time-seriesimage data (S101).

The association unit 110 transmits sameness determination informationincluding a result of association between the same objects to theattribute change detection unit 120.

The attribute change detection unit 120 detects an attribute change ofthe same object between pieces of time-series image data using thesameness determination information of the object. The attribute changedetection unit 120 transmits the attribute change detection result ofthe same object to the suspicious object detection unit 130.

If the attribute change detection unit 120 does not detect an attributechange of the same object between pieces of time-series image data (Noin S102), the flow illustrated in FIG. 2 ends.

On the other hand, if the attribute change detection unit 120 detects anattribute change of the same object between pieces of time-series imagedata (Yes in S102), the suspicious object detection unit 130 detects anobject whose attribute has changed as a suspicious object (S103).

Thus, the operation flow of the monitoring device 100 according toExample embodiment 1 ends.

Effects of Present Example Embodiment

According to the configuration of Example embodiment 1, the associationunit 110 associates the same objects with each other among multipleobjects detected from time-series image data. The attribute changedetection unit 120 detects a change in the attribute of at least one ofthe same object and accessory from the time-series image data. Thesuspicious object detection unit 130 detects a suspicious object on thebasis of a change in attribute. Hence, the monitoring device 100according to Example embodiment 1 can detect a suspicious object thatcannot be determined from the behavior of the object.

Example Embodiment 2

In Example embodiment 2, a configuration in which the same object isassociated and an attribute change is further detected using attributedata stored in an attribute data storage unit will be described.

(Monitoring Device 200)

FIG. 3 is a block diagram illustrating a configuration of a monitoringdevice 200 according to Example embodiment 2. As illustrated in FIG. 3,the monitoring device 200 includes a suspicious object detection unit210, an object detection unit 220, an attribute extraction unit 230, anassociation unit 240, an attribute change detection unit 250, and anattribute data storage unit 270. In the following description, a casewhere the object is a person will be mainly described, but the objectdoes not necessarily need to be a person. For example, the monitoringdevice 200 may be a system that monitors a vehicle, and the object maybe a vehicle.

(Object Detection Unit 220)

The object detection unit 220 acquires time-series image data. Forexample, the object detection unit 220 acquires moving image datacaptured by one or multiple monitoring cameras (not illustrated) fromthe monitoring cameras as time-series image data.

The object detection unit 220 detects multiple objects from the receivedtime-series image data. In Example embodiment 2, the object detectionunit 220 uses a discriminator having learned features of an object to bedetected, such as a person or an accessory, to detect the object fromeach piece of image data and extracts a region of the object.

The object detection unit 220 adds a label (hereinafter referred to asperson label) to each person detected from each piece of image data.Then, the object detection unit 220 transmits the image data of theregion of the person extracted from each image data to the attributeextraction unit 230 together with the person label.

The object detection unit 220 detects an accessory of the object. Forexample, the object detection unit 220 detects an object that is inclose contact with a person and has a large degree of overlap with theregion of the person as an accessory. The object detection unit 220 addsa label (hereinafter referred to as accessory label) to each detectedaccessory, and associates the accessory label with the above-describedperson label.

Alternatively, the object detection unit 220 may detect an object thatis separated from a person but is in the vicinity of the person as anaccessory, and the person and the accessory may be associated with eachother. At this time, the object detection unit 220 may also consider thepositional relationship between the person and the accessory.Alternatively, when only one person is near an object other than aperson, the object detection unit 220 may associate the object with theperson as an accessory.

Alternatively, the object detection unit 220 may associate a person withan accessory in consideration of reliability indicating the certainty ofassociation. For example, when there are multiple persons near anobject, the object detection unit 220 associates each person with theaccessory, and obtains the reliability of the association. Note,however, that when there are multiple persons near the accessory, thereliability of association between each person and the accessory islower than when only one person exists near the accessory. The personand the accessory may be associated not by the object detection unit 220but by the attribute extraction unit 230 described later.

The object detection unit 220 transmits position information of a personor an accessory including position coordinates (e.g., positioncoordinates of four corners of a region of a person or an accessory) ofa person or an accessory detected from each piece of image data to theassociation unit 240 together with a person or accessory label.

(Attribute Extraction Unit 230)

The attribute extraction unit 230 receives the image data of the regionof the person and the person label, and the image data of the region ofthe accessory and the label of the accessory from the object detectionunit 220. The attribute extraction unit 230 extracts the attributeinformation of the person from the image data of the region of theperson. For example, the attribute extraction unit 230 extractsinformation regarding the face, iris, gait, and hairstyle of the personfrom the image data of the region of the person as the attributeinformation of the person.

The attribute extraction unit 230 further extracts attribute informationindicating the attribute of the accessory from the image data of theregion of the accessory. The attribute extraction unit 230 storesattribute data in which the extracted attribute information of theperson and attribute information of the accessory are associated with aperson label received from the object detection unit 220 in theattribute data storage unit 270.

(Association Unit 240)

The association unit 240 associates the same persons with each otheramong multiple persons detected from time-series image data. Theassociation unit 240 is an example of association means.

Specifically, the association unit 240 receives, from the objectdetection unit 220, the position information of the person indicatingthe position coordinates of the region of the person in the image dataand the person label of the person.

The association unit 240 refers to the attribute data storage unit 270and acquires attribute data related to the person label received fromthe object detection unit 220. Attribute data associates attributeinformation of a person and attribute information of an accessory with aperson label.

Then, the association unit 240 associates the same objects with eachother among multiple objects detected from time-series image data, onthe basis of at least one of attribute information and positioninformation of the objects.

In one example, the association unit 240 associates the same personswith each other among multiple persons detected from time-series imagedata, on the basis of position information of the person received fromthe object detection unit 220. Specifically, the association unit 240tracks a person using position information of the person. Theassociation unit 240 associates the same persons with each other amongpersons detected from time-series image data by determining the samenessof the person detected from the time-series image data on the basis ofthe tracking result of the person (including trajectory of person).

Alternatively, the association unit 240 may associate the same personswith each other among multiple persons detected from time-series imagedata, on the basis of attribute information of the person included inattribute data. Specifically, the association unit 240 determines thesameness of the person by performing face matching or iris matchingusing the feature of the face or iris pattern of the person as theattribute information of the person. The association unit 240 associatesthe same person among all persons detected from the time-series imagedata on the basis of the sameness determination result of the person.

The association unit 240 may associate persons by using only one pieceof attribute information (e.g., facial feature, pattern of iris, and thelike of person).

Alternatively, when the attribute information of the person includesmultiple elements (e.g., face, iris, gait, body shape, and hairstyle),the association unit 240 may associate persons using all or some of themultiple elements of the attribute information. In this case, theassociation unit 240 weights the person sameness determination resultsthat are based on the multiple elements of the attribute information.Then, the association unit 240 associates the same person by integratingthe weighted person sameness determination results.

Alternatively, the association unit 240 may associate the same personswith each other among multiple persons detected from time-series imagedata on the basis of both attribute information and position informationof the object. For example, the association unit 240 associates the sameperson by weighting and integrating the person sameness determinationresult based on the position information of the person and the personsameness determination result obtained based on the attributeinformation of the person.

More generally, the association unit 240 obtains a likelihoodrepresenting the likelihood of being the same person for each attributeon the basis of each piece of attribute information. The associationunit 240 integrates the obtained likelihoods of attributes. Then, theassociation unit 240 determines that multiple persons are the sameperson when the integrated likelihood exceeds a certain threshold.Alternatively, the association unit 240 may learn a function forobtaining an integrated likelihood using multiple pieces of attributeinformation as inputs, by using a neural network or the like. In thiscase, the association unit 240 can directly obtain the integratedlikelihood from the multiple pieces of attribute information using thefunction obtained by learning. When the attribute extraction unit 230cannot extract some of the attribute information of the person, theassociation unit 240 determines the sameness of the person using onlythe extracted attribute information.

The association unit 240 may also obtain the reliability of the samenessdetermination. The reliability of the sameness determination depends onthe attribute information of the person used by the association unit 240and the distribution situation of other persons around the person.

Alternatively, the association unit 240 may indirectly associate thesame person by using distribution information of other persons,environment constraint information, and the like. One example is thecase described in Example embodiment 1 above in which all persons exceptonly one person are identified among multiple persons detected by theobject detection unit 220. In such a case, the association unit 240associates the remaining person as the same person without identifyingthe one person.

The association unit 240 associates the same person among personsdetected from time-series image data by any of the methods describedabove. The association unit 240 transmits information (e.g., group ID)indicating a group of person labels related to the same person to theattribute change detection unit 250.

For example, it is assumed that a person a and a person b detected fromdifferent pieces of image data are the same. In this case, theassociation unit 240 puts the person label added to the person a and theperson label added to the person b in a group of same persons. Then, theassociation unit 240 assigns the same group ID to multiple person labelsforming the group of same persons, and transmits the group ID to theattribute change detection unit 250 as information indicating the groupof same persons.

(Attribute Change Detection Unit 250)

The attribute change detection unit 250 receives, from the associationunit 240, information indicating the group of same persons as a resultof the association of persons by the association unit 240.

The attribute change detection unit 250 detects an attribute change ofthe same object (person in this example) between pieces of time-seriesimage data, on the basis of the result of the association of persons bythe association unit 240.

Specifically, the attribute change detection unit 250 acquires multiplepieces of attribute data related to multiple person labels associatedwith one group ID from the attribute data storage unit 270, using theinformation indicating the group of same persons. As described above,attribute data of a person includes attribute information of the personand attribute information of the accessory.

The attribute change detection unit 250 determines whether theattributes of the same persons match. That is, the attribute changedetection unit 250 determines whether the attribute of a personassociated with each person label and the attribute of a personassociated with another person label match. For example, multiple personlabels related to the same persons (i.e., multiple person labelsassociated with one group ID) are defined as person label L₁, personlabel L₂, . . . , person label L_(n). The attribute change detectionunit 250 determines whether the attribute of a person associated withthe person label L₁ and the attribute of a person associated with aperson label L_(k) (k=2, 3, . . . , n) match.

When the attributes of the same persons or the attributes of theaccessories of the same person do not match, the attribute changedetection unit 250 determines that the attribute of at least one of theperson and the accessory has changed between the pieces of time-seriesimage data. Hereinafter, a change in the attribute of a person or theattribute of an accessory of a person is referred to as an attributechange of the person or simply an attribute change.

Note, however, that as described in Example embodiment 1 above, thesimilarity between the attributes of the same persons may temporarilydecrease due to factors such as an environmental change. For thisreason, after the similarity between the attributes of the same personsdecreases, the attribute change detection unit 250 may detect anattribute change after confirming that the value of the attribute of theperson is stable.

The attribute extraction unit 230 cannot always extract all theattribute information of a person from image data. For this reason, theattribute change detection unit 250 detects an attribute change only onthe basis of attribute information that can be extracted by theattribute extraction unit 230. When the same person has multipleattributes, the attribute change detection unit 250 detects a change inat least one attribute among the multiple attributes as an attributechange. Alternatively, the attribute extraction unit 230 may detect achange, disappearance, or appearance of a specific accessory of a personas an attribute change. For example, the attribute change detection unit250 may detect, as an attribute change, that the presence of anaccessory is no longer detected from a certain point of time, orconversely, that an accessory is detected from a certain point of time.

The attribute change detection unit 250 transmits the attribute changedetection result to the suspicious object detection unit 210. Theattribute change detection result includes the type of the changedattribute, the content of the attribute change (e.g., change in color,number, size, shape, pattern, texture, number, position, or the like ofaccessory), and the sameness determination information of the personwhose attribute has changed. The sameness determination information mayalso include reliability information indicating the certainty of thesameness.

(Suspicious Object Detection Unit 210)

The suspicious object detection unit 210 detects a suspicious object onthe basis of the attribute change detection result. The suspiciousobject detection unit 210 is an example of suspicious object detectionmeans.

Specifically, the suspicious object detection unit 210 receives anattribute change detection result from the attribute change detectionunit 250. The suspicious object detection unit 210 detects a personwhose attribute has changed as a suspicious object. For example, in acase where the attribute of an accessory possessed, accompanied,carried, held, worn, pushed, or pulled by a person changes, thesuspicious object detection unit 210 detects the person as a suspiciousobject. For example, the suspicious object detection unit 210 detects aperson whose facial feature has changed as a suspicious object.

The suspicious object detection unit 210 may give notice that asuspicious object has been detected. For example, the suspicious objectdetection unit 210 may output an alert or may notify a terminalpossessed by the security guard. The suspicious object detection unit210 may output the content (e.g., change in color of accessory such asclothing) of the detected attribute change and image data of the personwhose attribute has changed to a display device (not illustrated).

Similarly to the suspicious object detection unit 130 according toExample embodiment 1 above, the suspicious object detection unit 210 maycalculate the “degree of suspicion” that is a numerical valuerepresenting the degree (level) of suspicion of the object by matchingthe detected attribute change of the object with a predeterminedattribute change pattern. Then, the suspicious object detection unit 210may accumulate the “degree of suspicion” in time series and output analert when the accumulated value exceeds a threshold.

(Operation Flow of Monitoring Device 200)

An operation flow of the monitoring device 200 according to Exampleembodiment 2 will be described with reference to FIG. 4. FIG. 4 is aflowchart illustrating an operation flow of the monitoring device 200.

As illustrated in FIG. 4, the object detection unit 220 acquirestime-series image data.

In Example embodiment 2, the object detection unit 220 receives imageframes (hereinafter referred to as image data) of moving image datacaptured by the monitoring camera one by one in real time. The objectdetection unit 220 detects one or multiple objects (person or accessoryin this example) from the received single piece of image data (S201).

The object detection unit 220 transmits image data of a region of eachperson and accessory in the image data to the attribute extraction unit230 together with identification information (hereinafter referred to asperson label and accessory label) of each person and accessory. Theobject detection unit 220 transmits position information of each personto the association unit 240 together with the person label assigned toeach person.

The attribute extraction unit 230 receives the image data of the regionof the person and the accessory together with the person label and theaccessory label from the object detection unit 220. The attributeextraction unit 230 extracts attribute information of the person andattribute information of the accessory of the person from the image dataof the region of the person and the accessory (S202).

The attribute extraction unit 230 stores the extracted attributeinformation of the person and the accessory in the attribute datastorage unit 270 as attribute data in association with the person labelreceived from the object detection unit 220. Hereinafter, when simplydescribed as attribute information of a person, attribute information ofthe accessory is also included.

The association unit 240 receives position information of the persontogether with the person label from the object detection unit 220.

The association unit 240 associates the same person on the basis of atleast one of the attribute information and the position information ofthe person (S203).

The association unit 240 transmits information (i.e., group ID)indicating a group of multiple person labels related to the same personto the attribute change detection unit 250.

The attribute change detection unit 250 receives the group ID indicatinga group of multiple person labels related to the same person from theassociation unit 240. The attribute change detection unit 250 acquiresmultiple pieces of attribute data related to the same person from theattribute data storage unit 270 using the multiple person labels relatedto the same person.

Then, the attribute change detection unit 250 determines whether theattribute of the same person changes using the multiple pieces ofattribute data acquired from the attribute data storage unit 270 (S204).

For example, the association unit 240 can use the similarity or thedistance between the attributes of the same person as a scale fordetermination in step S204.

If the attribute of the same person has not changed (No in S204), theoperation flow returns to step S201.

On the other hand, if the attribute of the same person has changed (Yesin S204), the suspicious object detection unit 210 detects the personwhose attribute has changed as a suspicious object (S205).

If the monitoring of the object (person) by the monitoring device 200 iscontinued (Yes in S206), the operation flow returns to step S201. If themonitoring of the object (person) by the monitoring device 200 is notcontinued (No in S206), the operation flow ends. The monitoring device200 may be controlled to continue monitoring only in a certain time zoneby a timer, for example.

Effects of Present Example Embodiment

According to the configuration of the present example embodiment, theassociation unit 240 associates the same objects with each other amongmultiple objects detected from time-series image data. The attributechange detection unit 250 detects a change the attribute of at least oneof the same object and accessory from the time-series image data. Thesuspicious object detection unit 210 detects a suspicious object on thebasis of a change in attribute. Hence, it is possible to detect asuspicious object that cannot be determined from the behavior of theobject.

Example Embodiment 3

In Example embodiment 3, a configuration for verifying the authenticityof a detected attribute change will be described. In Example embodiment3, a true attribute change means an actual attribute change. A falseattribute change means an attribute change that is erroneously detectedeven though it has not actually occurred. In the following description,a case where the object is a person will be mainly described, but theobject may be an object other than a person such as a vehicle.

(Monitoring Device 300)

FIG. 5 is a block diagram illustrating a configuration of a monitoringdevice 300 according to Example embodiment 3. As illustrated in FIG. 5,the monitoring device 300 includes a suspicious object detection unit210, an object detection unit 220, an attribute extraction unit 230, anassociation unit 240, an attribute change detection unit 250, and anattribute data storage unit 270. Additionally, in the monitoring device300, the suspicious object detection unit 210 includes an attributechange verification unit 290.

The configuration of the monitoring device 300 according to Exampleembodiment 3 is obtained by adding the attribute change verificationunit 290 to the configuration of the monitoring device 200 according toExample embodiment 2 above. In Example embodiment 3, only the attributechange verification unit 290 will be described below.

(Attribute Change Verification Unit 290)

The attribute change verification unit 290 verifies the authenticity ofan attribute change of an object (person in this example) detected bythe attribute change detection unit 250. The attribute changeverification unit 290 is an example of attribute change verificationmeans.

Specifically, the attribute change verification unit 290 receives, fromthe attribute change detection unit 250, an attribute change detectionresult and information (e.g., frame number of image data) for specifyingthe image data in which the attribute change is detected.

The attribute change verification unit 290 specifies the image data inwhich the attribute change is detected among pieces of time-series imagedata by using the information (e.g., frame number of image data) forspecifying the image data in which the attribute change is detected.

The attribute change verification unit 290 acquires, from thetime-series image data, one or multiple pieces of image data(hereinafter referred to as verification image data) corresponding toimage data in which the attribute change is detected and image frameswithin a predetermined time range before and after the image data.

The attribute change verification unit 290 verifies the authenticity ofthe attribute change detected by the attribute change detection unit 250using the acquired verification image data.

Here, there are several reasons why a false attribute change isdetected. One example is that the association unit 240 has failed inassociating the same person. In particular, in a case where a largenumber of persons are present at the same time (i.e., in the same imagedata), in a case where the attribute extraction unit 230 cannotaccurately extract the attributes of the person and the accessory, or ina case where the association unit 240 performs the samenessdetermination only on the basis of the position information of theperson, the association unit 240 is likely to fail to correctlyassociate the same person. The attribute extraction unit 230 may fail toextract correct attribute information of a person or an accessory due tothe influence of another person, such as the person being shielded byanother person. As described above, the attribute change detection unit250 may erroneously detect an attribute change due to erroneousassociation of the same person by the association unit 240 or extractionof the attribute information of an erroneous person or accessory by theattribute extraction unit 230. These are false attribute changes.

When the attribute change is false, the attribute change verificationunit 290 removes (hereinafter sometimes referred to as filtering orfiltering out) the attribute change detection result by the attributechange detection unit 250.

Hence, the suspicious object detection unit 210 detects a person as asuspicious object when the change in the attribute of the person istrue, and does not detect a person as a suspicious object when thechange in the attribute of the person is false. The determination oftrue or false does not necessarily need to be a binary decision, and thesuspicious object detection unit 210 may calculate an index indicatingthe degree of trueness. Then, when the calculated value of the indexexceeds a threshold designated by the user, the suspicious objectdetection unit 210 may consider that the detected attribute change istrue and detect the person as a suspicious object.

(Example of False Attribute Change)

In any of the following cases, the attribute change verification unit290 determines that the detected attribute change is false. Note,however, that the cases described below are merely examples.

(Case 1) Failure in Detection of Person and Resulting Failure inExtraction of Attribute Information

When a person is hidden by a product shelf (container), another person,or the like, the object detection unit 220 may not be able to accuratelyextract the region of the person from image data. In this case, theregion of the person extracted by the object detection unit 220 mayinclude a background or a region of another object. For this reason, theattribute extraction unit 230 cannot correctly extract attributeinformation of the person from the region of the person extracted by theobject detection unit 220.

In this case, the attribute change verification unit 290 separates theverification image data into the region of the person (foreground) andthe background by the background subtraction method, for example, anddistinguishes the person from the other regions. Note, however, that inorder to use the background subtraction method, time-series image dataneeds to be captured by a fixed camera.

The attribute change verification unit 290 compares the backgroundregion (region determined not to be person) in the verification imagedata with the region of the person detected by the object detection unit220. Then, when reliability of the attribute extracted from the regionof the person or the region of the accessory is low because a part ofthe detected region of the person is shielded by a background regionsuch as a shelf, the attribute change verification unit 290 determinesthat the detected attribute change is false.

Alternatively, the attribute change verification unit 290 verifiesoverlap between a person and another person in verification image data.When the overlap between the person and the another person is large, apossibility that the person is shielded is high. In that case, theattribute change verification unit 290 determines that the detectedattribute change is false. Here, the attribute change verification unit290 preferably checks whether the another person overlapping the personexists, including not only the image data related to the image frame inwhich the person is detected but also the image frame temporally closeto the image frame. As a result, even when the another person is notdetected in the image data related to the image frame in which theperson is detected, the attribute change verification unit 290 may beable to detect the another person from the image frame temporally closeto the image frame.

(Case 2) Change in Posture of Person

When a person tilts his/her head, crouches, or leans forward, theattribute information (e.g., face and gait of person) of the personapparently changes on the image data. In this case, the association unit240 may fail to associate the same person or may fail to extract theattribute of a person.

The attribute change verification unit 290 detects a change in theposture of a person by analyzing verification image data. When a changein the posture of a person is detected from verification image data,even if an attribute change is detected, the attribute changeverification unit 290 determines that the attribute change is false forthe attribute affected by the posture change.

(Case 3) Variation of Illumination

When the brightness or color of the illumination changes, the attributesof an accessory such as the color of the person's clothing apparentlychange. In this case, the attribute change detection unit 250 mayerroneously detect an attribute change.

The attribute change verification unit 290 detects a change in at leastone of brightness and color of the background (or specific region) fromthe verification image data. Then, in a case where at least one of thebrightness and the color of the background (or specific region) in theverification image data has changed, the attribute change verificationunit 290 determines that the illumination has changed. When theattribute change detection unit 250 detects an attribute change affectedby a variation (e.g., brightness) of the illumination, the attributechange verification unit 290 determines that the detected attributechange is false.

Alternatively, the attribute change verification unit 290 may alsoconsider the reliability of association of the same person by theassociation unit 240. For example, in image frames temporally before andafter the image frame in which the attribute change is detected, theattribute change verification unit 290 checks the reliability ofassociation of the same person by the association unit 240. When thereliability of association of the same person by the association unit240 is low, the attribute change verification unit 290 may determinethat the detected attribute change is a false attribute change. Whencalculating the reliability of association between a person and anaccessory, the object detection unit 220 (or attribute extraction unit230) may also consider this reliability to determine the authenticity ofthe attribute change.

(Operation Flow of Monitoring Device 300)

An operation flow of the monitoring device 300 according to Exampleembodiment 3 will be described with reference to FIG. 6. FIG. 6 is aflowchart illustrating an operation flow of the monitoring device 300.

Steps S301 to S304 illustrated in FIG. 6 are the same as steps S201 toS204 of the operation flow of the monitoring device 200 described inExample embodiment 2 above. Here, an operation flow from step S304 willbe described.

In step S304, the attribute change detection unit 250 determines whetherthe attribute of the same person changes between pieces of time-seriesimage data.

When the attribute of the same person changes between pieces oftime-series image data (Yes in S304), the attribute change verificationunit 290 verifies the authenticity of the attribute change detected bythe attribute change detection unit 250 (S305).

If the attribute change is false (No in S306), the attribute changeverification unit 290 filters out the attribute change detection resultof the object. Then, the operation flow returns to step S301.

On the other hand, if the attribute change is true (Yes in S306), thesuspicious object detection unit 210 detects the person whose attributehas changed as a suspicious object (S307).

If the monitoring of the object (person) by the monitoring device 300 iscontinued (Yes in S308), the operation flow returns to step S301. If themonitoring of the object (person) by the monitoring device 300 is notcontinued (No in S308), the operation flow ends.

The monitoring device 300 may be controlled to continue monitoring onlyin a certain time zone by a timer, for example.

Effects of Present Example Embodiment

According to the configuration of the present example embodiment, theassociation unit 240 associates the same objects with each other amongmultiple objects detected from time-series image data. The attributechange detection unit 250 detects a change in the attribute of at leastone of the same object and accessory from the time-series image data.The suspicious object detection unit 210 detects a suspicious object onthe basis of a change in attribute. Hence, it is possible to detect asuspicious object that cannot be determined from the behavior of theobject.

Further, the attribute change verification unit 290 verifies theauthenticity of the attribute change of the object detected by theattribute change detection unit 250. The attribute change verificationunit 290 filters out a false attribute change. The suspicious objectdetection unit 210 detects the same object as a suspicious object if theattribute change is true, and does not detect the same object as asuspicious object if the attribute change is false. For this reason, thesuspicious object can be more accurately detected based on the trueattribute change.

Example Embodiment 4

In Example embodiment 4, a configuration for analyzing a contextassociated with an attribute change will be described. In Exampleembodiment 4, an associated context means at least one of an environmentthat affects the attribute change and an event that occurs inassociation with the attribute change. In the following description, acase where the object is a person will be mainly described, but theobject may be an object other than a person such as a vehicle.

(Monitoring Device 400)

FIG. 7 is a block diagram illustrating a configuration of a monitoringdevice 400 according to Example embodiment 4. As illustrated in FIG. 7,the monitoring device 400 includes a suspicious object detection unit210, an object detection unit 220, an attribute extraction unit 230, anassociation unit 240, an attribute change detection unit 250, and anattribute data storage unit 270. Additionally, in the monitoring device400, the suspicious object detection unit 210 includes a contextanalysis unit 280.

The configuration of the monitoring device 400 according to Exampleembodiment 4 is obtained by adding the context analysis unit 280 to theconfiguration of the monitoring device 200 according to Exampleembodiment 2 above. In Example embodiment 4, only the context analysisunit 280 will be described below.

(Context Analysis Unit 280)

The context analysis unit 280 analyzes the context associated with anattribute change of an object. The context analysis unit 280 is anexample of context analysis means.

Specifically, the context analysis unit 280 receives, from the attributechange detection unit 250, an attribute change detection result of aperson or an accessory (one example of object) and information (e.g.,frame number of image data) for specifying the image data in which theattribute change is detected.

The context analysis unit 280 specifies the image data in which theattribute change is detected among pieces of time-series image data byusing the information (e.g., frame number of image data) for specifyingthe image data in which the attribute change is detected. Then, thecontext analysis unit 280 acquires, from the time-series image data, oneor multiple pieces of image data (hereinafter referred to as contextanalysis image data) corresponding to image data in which the attributechange is detected and image frames within a predetermined time rangebefore and after the image data.

The context analysis unit 280 analyzes the acquired context analysisimage data to specify the context associated with the attribute change.Specifically, the context analysis unit 280 specifies at least one of anenvironment (temperature, time zone, weather, humidity, brightness,human density, and the like) that affects the attribute change and anevent (person's behavior, incident, accident, traffic condition, and thelike) that has occurred in association with the attribute change.

Alternatively, the context analysis unit 280 may acquire informationsuch as the temperature, weather, humidity, brightness, and humandensity using a sensor or the like other than the imaging sensor.Alternatively, the context analysis unit 280 may also acquireinformation such as an incident, an accident, and a traffic situationfrom another external information source.

The context analysis unit 280 searches for exception information relatedto a combination of an attribute change and an associated context.Exception information may be stored in advance in a server on a networkconnected to the monitoring device 400.

Exception information is information describing a specific attributechange that is not suspicious only in a specific context. For example,when moving from the outside where it is cold to a warm room, a persontakes off his/her jacket and hat. In the above example, the movement ofthe person from the outside to the room is a specific event, and thechange in the clothing of the person is a specific attribute change.

Exception information associates information of a specific context withinformation of a specific attribute change. Exception information ispreferably present for each type of object and for each type ofaccessory. Exception information may be stored in advance in anexception information storage unit (not illustrated).

In one modification, exception information may be generated by thecontext analysis unit 280. Specifically, in a case where a specificattribute change is frequently (e.g., equal to or more than thresholdnumber of times) detected by the attribute change detection unit 250 ina specific context (at least one of specific environment and specificevent), the context analysis unit 280 generates exception information inwhich the specific attribute change is associated with the specificcontext. The context analysis unit 280 may transmit the generatedexception information to a server that stores exception information, ormay store the generated exception information in an exceptioninformation storage unit (not illustrated).

When exception information, related to the combination of the attributechange and the associated context, exists, the context analysis unit 280filters out the attribute change detection result of the object.

That is, the suspicious object detection unit 210 detects the sameobject as a suspicious object in a case where the combination of theattribute change and the associated context does not correspond to aspecific exception, and does not detect the same object as a suspiciousobject in a case where the combination of the attribute change and theassociated context corresponds to a specific exception.

Note that the suspicious object detection unit 210 according to Exampleembodiment 4 may restrict filtering out the attribute change detectionresult by the context analysis unit 280 in a special situation such as acase where a crime occurs. This is because special situations may occureven in situations corresponding to specific exceptions.

(Specific Exception)

FIG. 8 is a diagram illustrating an example of the exception informationdescribed above. The following Examples 1 to 3 correspond to No. 1 toNo. 3 illustrated in FIG. 8. The following Examples 1 to 3 will bedescribed in order.

Example 1

In a locker room at work, a person changes into a uniform. In thisexample, the change in clothing corresponds to a specific attributechange in a specific exception, and the locker room corresponds to aspecific environment in a specific exception. On the other hand, at theentrance of a residence building, the clothing (attribute information)of a person is usually not likely to change to a uniform of a maildelivery person or a home delivery agent. Hence, the combination of theattribute change (change in clothing) of the accessory and theenvironment (entrance of residence building) in the latter example doesnot correspond to a specific exception.

Example 2

In a railway station, since luggage is loaded into a locker, the numberof luggage (accessory) of a person changes. When luggage (one example ofaccessory) carried on the back of a person is held by hand, the positionof the luggage changes. In these examples, a change in the number orlocation of a person's luggage corresponds to a specific attributechange in a specific exception, and loading of luggage or change in theway of carrying the luggage corresponds to a specific event in aspecific exception.

Example 3

When moving from the outside where it is cold to a warm room, a persontakes off his/her coat and hat. In the above example, the change in theclothing of the person is a specific attribute change in a specificexception, and the movement of the person from the outside to the roomis a specific event in a specific exception. In the present example, itis preferable that exception information further includes informationregarding an environment that affects the specific attribute change,such as information on the outside temperature, humidity, and season. Asa result, the context analysis unit 280 can more accurately determinethe specific exception on the basis of the exception information.

(Operation Flow of Monitoring Device 300)

An operation flow of the monitoring device 400 according to Exampleembodiment 4 will be described with reference to FIG. 9. FIG. 9 is aflowchart illustrating an operation flow of the monitoring device 400.In the operation flow of the monitoring device 400 illustrated in FIG.9, the flow from step S401 to step S404 is the same as the flow fromstep S201 to step S204 of the operation flow of the monitoring device200 illustrated in FIG. 4 described in Example embodiment 2 above.

First, the object detection unit 220 acquires time-series image data. InExample embodiment 4, too, the object detection unit 220 receives imageframes (hereinafter referred to as image data) of moving image datacaptured by the monitoring camera one by one in real time. The objectdetection unit 220 detects objects (person and accessory in thisexample) from the received single piece of image data (S401).

The object detection unit 220 transmits image data of a region of aperson and an accessory in the image data to the attribute extractionunit 230 together with information (hereinafter referred to as personlabel and accessory label) for identifying the region of the person andthe accessory. The object detection unit 220 transmits positioninformation of the person detected from the image data to theassociation unit 240 together with the person label.

The attribute extraction unit 230 receives the image data of the regionof the person and the accessory together with the person label and theaccessory label from the object detection unit 220. The attributeextraction unit 230 extracts attribute information of the person andattribute information of the accessory from the image data of the regionof the person and the region of the accessory (S402). Hereinafter, whensimply described as attribute information of a person, attributeinformation of the accessory is also included.

The attribute extraction unit 230 stores the extracted attributeinformation of the person (including attribute information of accessory)in association with the person label in the attribute data storage unit270 as attribute data.

The association unit 240 receives position information of the persontogether with the person label from the object detection unit 220.

The association unit 240 associates the same person on the basis of theposition information of the person received from the object detectionunit 220 (S403).

The association unit 240 generates a group of person labels related tothe same person. The association unit 240 transmits information (e.g.,multiple person labels associated with one group ID) indicating a groupof the same persons to the attribute change detection unit 250.

The attribute change detection unit 250 receives the informationindicating the group of the same persons from the association unit 240.

The attribute change detection unit 250 acquires multiple pieces ofattribute data associated to multiple person labels related to the sameperson in the attribute data storage unit 270. Then, the attributechange detection unit 250 determines whether the pieces of attributeinformation included in the multiple pieces of acquired attribute datamatch (S404).

In step S404, the association unit 240 can use the similarity ordistance between pieces of attribute information, for example, as ascale for determination in step S404.

If the pieces of attribute information of the person associated to themultiple person labels related to the same person match, the attributechange detection unit 250 determines that the attribute of the sameperson has not changed between pieces of time-series image data (No inS404). In this case, the operation flow returns to step S401.

On the other hand, if at least one of the elements of the attributeinformation of the person associated to the multiple person labelsrelated to the same person does not match, the attribute changedetection unit 250 determines that the attribute of the same person haschanged between pieces of time-series image data (Yes in S404). That is,the attribute change detection unit 250 detects an attribute change.

In Example embodiment 4, the attribute change detection unit 250transmits the attribute change detection result to the context analysisunit 280 together with information (e.g., frame number of image data)for specifying the image data in which the attribute change is detected.

The context analysis unit 280 receives, from the attribute changedetection unit 250, the attribute change detection result andinformation (e.g., frame number of image data) for specifying the imagedata in which the attribute change is detected.

The context analysis unit 280 acquires context analysis image datarelated to the image data in which the attribute change is detected fromthe time-series image data. Then, the context analysis unit 280 analyzesthe context associated with the attribute change using the contextanalysis image data (S405).

Specifically, in step S405, the context analysis unit 280 specifies atleast one of an environment (temperature, humidity, brightness, timezone, weather, human density, and the like) that affects the attributechange and an event (person's behavior, incident, accident, trafficcondition, and the like) that has occurred in association with theattribute change.

The context analysis unit 280 searches for exception information relatedto the combination of the detected attribute change and the associatedcontext (S406).

If there is exception information related to the combination of thedetected attribute change and the associated context (Yes in S406), thecontext analysis unit 280 filters out the attribute change detectionresult of the object. Then, the operation flow returns to step S401.

On the other hand, if there is no exception information related to thecombination of the attribute change and the associated context (No inS406), the suspicious object detection unit 210 detects the person whoseattribute has changed as a suspicious object (S407).

If the monitoring of the object (person) by the monitoring device 400 iscontinued (Yes in S408), the operation flow returns to step S401. If themonitoring of the object (person) by the monitoring device 400 is notcontinued (No in S408), the operation flow ends. The monitoring device400 may be controlled to continue monitoring only in a certain time zoneby a timer, for example.

Effects of Present Example Embodiment

According to the configuration of Example embodiment 4, the associationunit 240 associates the same objects with each other among multipleobjects detected from time-series image data on the basis of at leastone of attribute information and position information of the multipleobjects, and the attribute change detection unit 250 detects anattribute change of the same object between pieces of the time-seriesimage data. Hence, it is possible to curb erroneous detection of anattribute change of a suspicious object.

Further, the context analysis unit 280 analyzes the context associatedwith the attribute change of the object. Of the attribute changesdetected by the attribute change detection unit 250, the contextanalysis unit 280 filters out the attribute change related to a specificexception on the basis of the analysis result of the context. As aresult, it is possible to prevent an object that is not suspicious frombeing erroneously detected.

(Modification)

The configurations described in Example embodiments 3 and 4 above can becombined. That is, the suspicious object detection unit 210 may includeboth the attribute change verification unit 290 and the context analysisunit 280.

In the present modification, the attribute change verification unit 290verifies the authenticity of an attribute change detected by theattribute change detection unit 250, and filters out (that is, removes)a false attribute change. When the combination of the (true) attributechange and the associated context corresponds to a specific exception,the context analysis unit 280 further filters out the attribute change.

According to the configuration of the present modification, both a falseattribute change and an attribute change corresponding to a specificexception are filtered out. For this reason, it is possible to morereliably prevent an object that is not suspicious from being erroneouslydetected.

Example Embodiment 5

Example embodiment 5 will be described with reference to FIG. 10.

FIG. 10 is a block diagram illustrating a configuration of a monitoringdevice 10 according to Example embodiment 5. As illustrated in FIG. 10,the monitoring device 10 includes an association unit 11, an attributechange detection unit 12, and a suspicious object detection unit 13.

The association unit 11 associates the same object among multipleobjects detected from time-series image data.

The attribute change detection unit 12 detects a change in the attributeof at least one of the same object and its accessory from thetime-series image data.

The suspicious object detection unit 13 detects a suspicious object onthe basis of a change in attribute.

With the above configuration, the monitoring device 10 according toExample embodiment 5 can detect a change in attribute between the timeswhen the object is detected for the objects associated as the sameobject, and thus can detect a suspicious object that cannot bedetermined from the behavior of the object.

Example Embodiment 6

Example embodiment 6 will be described below with reference to FIG. 11.

(Hardware Configuration)

Each component of the monitoring devices 10, 100, 200, 300, and 400described in Example embodiments 1 to 5 above represents a functionalunit block. Some or all of these components are implemented by aninformation processing apparatus 900 as illustrated in FIG. 11, forexample. FIG. 11 is a block diagram illustrating an example of ahardware configuration of the information processing apparatus 900according to Example embodiment 6.

As illustrated in FIG. 11, the information processing apparatus 900includes the following configuration as an example.

-   -   Central Processing Unit (CPU) 901    -   Read Only Memory (ROM) 902    -   Random Access Memory (RAM) 903    -   Program 904 loaded into RAM 903    -   Storage device 905 storing program 904    -   Drive device 907 that reads and writes recording medium 906    -   Communication interface 908 connected to communication network        909    -   Input/output interface 910 for inputting and outputting data    -   Bus 911 connecting each component

The components of the monitoring devices 10, 100, 200, 300, and 400described in Example embodiments 1 to 5 above are implemented by the CPU901 reading and executing the program 904 that implements thesefunctions. The program 904 for implementing the function of eachcomponent is stored in the storage device 905 or the ROM 902 in advance,for example, and the CPU 901 loads the program into the RAM 903 andexecutes the program as necessary. The program 904 may be supplied tothe CPU 901 through the communication network 909, or may be stored inadvance in the recording medium 906, and the drive device 907 may readthe program and supply the program to the CPU 901.

Effects of Present Example Embodiment

According to the configuration of the present example embodiment, themonitoring devices 10, 100, 200, 300, and 400 described in Exampleembodiments 1 to 5 above are implemented as hardware. Hence, effectssimilar to the effects described in Example embodiments 1 to 5 above canbe obtained.

While the present invention has been particularly shown and describedwith reference to exemplary embodiments thereof, the invention is notlimited to these embodiments. It will be understood by those of ordinaryskill in the art that various changes in form and details may be madetherein without departing from the spirit and scope of the presentinvention as defined by the claims.

INDUSTRIAL APPLICABILITY

The present invention can be used for detecting a suspicious person inan important facility such as an airport or a public facility. Thepresent invention may also be used for control (e.g., temperatureadjustment of an air conditioner, and the like) of a device using adetection result of an attribute change of an object.

This application is based upon and claims the benefit of priority fromJapanese patent application No. 2019-061223, filed on Mar. 27, 2019, thedisclosure of which is incorporated herein in its entirety by reference.

REFERENCE SIGNS LIST

-   100 monitoring device-   110 association unit-   120 attribute change detection unit-   130 suspicious object detection unit-   200 monitoring device-   210 suspicious object detection unit-   220 object detection unit-   230 attribute extraction unit-   240 association unit-   250 attribute change detection unit-   280 context analysis unit-   290 attribute change verification unit-   300 monitoring device-   400 monitoring device

What is claimed is:
 1. A monitoring device comprising: a memory; and atleast one processor coupled to the memory, the at least one processorperforming operations to: associate same object among multiple objectsdetected from time-series image data; detect an attribute change of atleast one of the same object and an accessory of the same object fromthe time-series image data; and detect a suspicious object based on theattribute change.
 2. The monitoring device according to claim 1, whereinthe at least one processor further performs operation to: verify theauthenticity of the attribute change, and detect the same object as thesuspicious object when the attribute change is true, and do not detectthe same object as the suspicious object when the attribute change isfalse.
 3. The monitoring device according to claim 1, wherein the atleast one processor further performs operation to: analyze a contextassociated with the attribute change, and detect the same object as asuspicious object when a combination of the attribute change and theassociated context does not correspond to a specific exception, and donot detect the same object as the suspicious object when the combinationof the attribute change and the associated context corresponds to thespecific exception.
 4. The monitoring device according to claim 3,wherein the associated context is at least one of an environment thataffects the attribute change and an event that occurs in associationwith the attribute change.
 5. The monitoring device according to claim1, wherein the at least one processor further performs operation to:associate the same object based on at least one of attribute informationand position information of the multiple objects.
 6. The monitoringdevice according to claim 5, wherein the object is a person, and anattribute of the person includes at least one of the face, the iris, thegait, the body shape, and the hairstyle of the person.
 7. The monitoringdevice according to claim 1, wherein an attribute of the accessoryincludes at least one of a color, a size, a shape, a pattern, a number,a position, and a texture of the accessory.
 8. The monitoring deviceaccording to claim 1, wherein the at least one processor furtherperforms operation to: output a detection result of the suspiciousobject.
 9. A suspicious object detecting method comprising: associatingsame object among a plurality of objects detected from time-series imagedata; detecting an attribute change of at least one of the same objectand an accessory of the same object from the time-series image data; anddetecting a suspicious object based on the attribute change.
 10. Anon-transitory computer-readable recording medium that records a programfor causing a computer to execute processing of: associating same objectamong multiple objects detected from time-series image data; detectingan attribute change of at least one of the same object and an accessoryof the same object from the time-series image data; and detecting asuspicious object based on the attribute change.